There exists a technique for enhancement of an image resolution that synthesizes a plurality of low-resolution images with a positional displacement having an identical scene photographed therein to generate a high-resolution image.
For example, a technique for enhancement of the image resolution that performs detection (motion estimation) of the positional displacement between the inputted low-resolution images with accuracy smaller than a pixel unit (sub-pixel accuracy), and estimates a high-resolution image x that minimizes an evaluation function g(x) represented by Equation (1) when synthesizing the low-resolution images is described in Non-patent literature 1.
                    [                  Numerical          ⁢                                          ⁢          equation          ⁢                                          ⁢          1                ]                                                                      g          ⁡                      (            x            )                          =                                            ∑                              ∀                n                                      ⁢                                                  ⁢                          (                                                                                                            y                      n                                        -                                                                  A                        n                                            ⁢                      x                                                                                        2                            )                                +                      λ            ⁢                                                        C                ⁡                                  (                  x                  )                                                                                                      (        1        )            
In the above-mentioned Equation (1), x denotes a high-resolution image, y denotes an input low-resolution image, A denotes an image transformation matrix including motions between the images, down-sampling and so forth, C is a high-pass filter, and λ is a pre-configured constant.
A first term in the right-hand side of the Equation 1 is a term indicative of an error between the input low-resolution image to be estimated from the high-resolution image x and the actually inputted low-resolution image, and a second term in the right-hand side is a normalization term founded on a condition that the high-resolution image to be generated is smooth and signifies an edge amount of the high-resolution image. For this, the high-resolution image x to be generated has an inclination that the edges thereof become clearer as a whole as the value of the constant λ is smaller, and to the contrary, the edges thereof becomes relatively blurred as the value of a constant λ is larger.
In general, such a process for enhancement of the image resolution is called a super-resolution process.
As a technology using this technique for enhancement of the image resolution so far, there exist the following technologies.
1. The technology of separating the photographed video image into a high-frequency component and a low-frequency component, applying the super-resolution process to the high-frequency region image, and combining the image generated by applying a super-resolution process to the high-frequency component image with the image generated by applying an interpolation and enlargement process to the low-frequency component image (Patent literature 1).
2. The technology of, for each object within the images, performing an image layout structure analysis such as features of the object and a relative positional relation among the objects, and performing correspondence of layout structure information, thereby to detect a position deviation amount among the frame images, and enhancing the resolution of the images (Patent literature 2).
Hereinafter, one example of the super-resolution processing device related to the present invention is shown.
FIG. 7 is a block diagram of a super-resolution processing device 10 for generating the high-resolution image by synthesizing a plurality of low-resolution images.
This super-resolution processing device includes motion estimating means 11 and high-resolution image estimating means 12.
The motion estimating means 11 receives a plurality of the low-resolution images as inputs and estimates motion of the pixel between a reference low-resolution image, which is subjected to the enhancement of the image resolution, and a to-be-referenced low-resolution image for each pixel thereof with sub-pixel accuracy, and outputs an estimation result.
The high-resolution image estimating means 12 receives the low-resolution images and the motion estimation results as inputs, and estimates and outputs the high-resolution image that minimizes an evaluation function represented by Equation (1) from these items of information.
By the way, as one of the processes for enhancing an image quality, there exists the image process called a beautiful skin process. The so-called beautiful skin process is a process of reproducing a skin of a subject figure beautifully and smoothly by paying attention to a face area, being a most noticeable part in a figure picture.
For example, Patent literature 3 detects a skin color of the subject figure from the inputted images, and synthesizes, for an area with a skin color similar to the detected skin color, the input image and the input image subjected to a smoothing process responding to likelihood of the above skin color (which is referred to as a color intensity of the skin in the Patent literature 3), thereby to smooth only the skin area. Additionally, in the following explanation, it is assumed that the skin color likelihood is called a skin color characteristic degree.
The patent literature 3 calculates a skin color characteristic degree hx (i) in a pixel position i as shown in Equation (2) using values L (i), a(i), and b(i) having a pixel value in the above pixel expressed with a Lab color system therein. Wherein, L′, a′, and b′ denote the center of gravity of the Lab value in the skin color area, and WL, Wa, and Wb denote a weight. The skin color characteristic degree hx has a value of 0.0 to 1.0, and it is meant that the skin color likelihood is yielded as the value of the skin color characteristic degree hx approaches 1.0.
                    [                  Numerical          ⁢                                          ⁢          equation          ⁢                                          ⁢          2                ]                                                                      hx          ⁡                      (            i            )                          =                  exp          [                      -                          {                                                                    (                                                                                            L                          ′                                                -                                                  L                          ⁡                                                      (                            i                            )                                                                                                                      W                        L                                                              )                                    2                                +                                                      (                                                                                            a                          ′                                                -                                                  a                          ⁡                                                      (                            i                            )                                                                                                                      W                        a                                                              )                                    2                                +                                                      (                                                                                            b                          ′                                                -                                                  b                          ⁡                                                      (                            i                            )                                                                                                                      W                        b                                                              )                                    2                                            }                                ]                                    (        2        )            
One example of the beautiful skin processing device related to the present invention is shown. FIG. 8 is a block diagram prepared based on description of the beautiful skin processing device related to the present invention.
The beautiful skin processing device related to the present invention includes skin color characteristic degree calculating means 21 and smoothing processing means 22. The skin color characteristic degree calculating means 21 detects the skin color of the subject figure within the inputted image, and calculates and outputs, for each pixel within the image, the skin color likelihood thereof. The smoothing processing means 22 generates the output image by synthesizing the input image and the input image generated in the inside that has been subjected to the smoothing process, using the skin color characteristic degree of each pixel calculated by the skin color characteristic degree calculating means 21.
The image processing device for executing both of the super-resolution process and the beautiful skin process can be realized by linking the related super-resolution processing device and beautiful skin processing device described above as shown in a block diagram of FIG. 9. In the image processing device shown in FIG. 9, at first, super-resolution processing device 10 generates one high-resolution image from a plurality of the low-resolution images using the related super-resolution processing technique. Next, beautiful skin processing device 20 subjects the generated high-resolution image to the beautiful skin process, and outputs the output images.